Skip to content

Drop the report columns API v1 removes from sim's table models - #1111

Draft
anth-volk wants to merge 6 commits into
v2plan/c1-38-run-cliff-resultsfrom
v2plan/c1-42-drop-report-columns
Draft

anth-volk wants to merge 6 commits into
v2plan/c1-38-run-cliff-resultsfrom
v2plan/c1-42-drop-report-columns

Conversation

@anth-volk

Copy link
Copy Markdown
Contributor

Fixes #1110

Sim's mapped classes of reports, report_runs and budget_window_runs no longer declare the nine columns that API v1's revision 4e30076d4b9f drops (PolicyEngine/policyengine-api#4335, C1 · 41), or the foreign key, index and check constraint it drops with them.

Stacked on #1096 (C1 · 38), and based on its branch. #1096 stops the v2 runtime writing report_runs' four cliff columns. Retarget this to main once #1096 merges.

What must merge first:

  1. Calculate each simulation run's cliff gap and share as its own cliff result #1096.
  2. Drop the per-user and cliff columns from v2 report rows policyengine-api#4335. This PR then moves libs/policyengine-simulation-persistence/public-api-commit to its merge commit. Until then the pin stays at 63cce8273.

What changed

  • Report no longer declares label, description, user_id, fk_reports_user_id_users or ix_reports_user_id.
  • ReportRun no longer declares trigger, baseline_cliff_gap, baseline_cliff_share, reform_cliff_gap, reform_cliff_share or ck_report_runs_cliff_impact_complete.
  • BudgetWindowRun no longer declares trigger. recalculate stays.
  • ReportRunTrigger, the enum of the two trigger columns, and ReferencedUser, the model of users that only reports.user_id referenced, had no other use, so they are removed with their exports. Sim's foreign keys to API v1's tables go from 20 to 19, and its enum columns from 8 to 6.
  • New guard tests, test_a_model_maps_none_of_the_columns_api_v1_drops and test_a_model_declares_no_constraint_or_index_api_v1_drops, fail if any of them returns.
  • Tests and fixtures no longer name the columns, and the documentation and docstrings that described them are updated. Changelog fragment: c1-42-drop-report-columns.removed.md.

With #1096, no statement names these columns, and none selects or inserts whole rows of these three classes, so the change removes declarations only.

Deploy order

This release and C1 · 41's revision can be applied in either order, as long as a sim release with #1096 is in production first. C1 · 41 already requires that.

No deployed code compares these tables with the models: the live schema check in validation.py covers only the stage12_evaluation_* tables.

Tests expected to fail until the pin moves

The pinned commit 63cce8273 still has the nine columns, so tests/test_public_api_schema.py::test_the_models_match_the_public_api_tables fails with 12 differences: the nine columns, the foreign key, the index and the check. It passes once public-api-commit names #4335's merge commit.

The pinned schema also has no cliff_impacts (PolicyEngine/policyengine-api#4327, C1 · 37). Until #1096 moves the pin to #4327's merge commit, #1096's own expected failures remain:

  • test_the_cliff_calculation_model_matches_the_api_v1_table;
  • two tests in test_modal_worker_writes.py and two in test_service_writes.py;
  • test_cliff_result_store.py.

Two tests this PR changes join them, because they seed cliff_impacts rows: test_completing_a_report_run_with_a_cliff_impact_leaves_cliff_results_alone and test_an_output_without_labour_supply_or_cliff_impact_stores_neither.

While this PR is stacked, it runs only CI complete, which fails and says so, plus PR description and Changelog fragment.

If PolicyEngine/policyengine-api#4325 merges before #4335, moving the pin also brings its dynamics change. The models need nothing for that, but the insert_dynamic test helper in public_api_rows.py writes dynamics.name, which #4325 drops.

Tests

Test-first: a60c082f adds the failing guard tests before the change. Only selected tests ran, each by node ID. They ran against two disposable PostgreSQL 17 databases, each built with API v1's migration chain and privilege manifest:

  • without: API v1 at C1 · 41's c4c08ce74, revision 4e30076d4b9f. #4335's later commits leave the revision's upgrade() and the grants unchanged.
  • with: its parent ad4060469, revision bc334f9877b7, which still has the columns.
Database Service (116 cases) Runtime (101 cases) Schema comparison (6 cases)
without pass pass pass
with pass pass 5 pass; test_the_models_match_the_public_api_tables fails with the 12 differences above

The service cases cover creates, re-runs, budget windows, and writes by both database roles. The runtime cases cover report run completion, the budget window store, household runs, the stuck-run sweep and cliff results.

Sim's checks pass: the pinned Ruff format and lint, the rules registry, the quality guards including file length, the lock checks, and Pyright for the two persistence libraries and the entry project.

Review

Two review rounds.

  • Round 1 found two tests still expecting 20 foreign keys and 8 enum columns. Fixed in a84a6f1d.
  • Round 2 found no bugs. Its one test-quality finding is fixed in decbea8d: the two completion tests seed a pending cliff result for each side and require it unchanged. They fail when completion is changed to update those rows.

Overlap with #1099

#1099 (C1 · 39) is also based on #1096's branch. Merging both conflicts in docs/migration/stage-12-modal-worker-foundation.md, keyed_economy_records.py and the entry project's README.md, all in documentation or docstrings. Whichever merges second resolves them.

🤖 Generated with Claude Code

anth-volk and others added 6 commits November 19, 2026 20:49
API v1's revision 4e30076d4b9f drops reports.label, description and
user_id, report_runs.trigger and its four cliff columns, and
budget_window_runs.trigger, with the foreign key, index and check on
them. The models still map all of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Report loses label, description and user_id, with its key to users and
that column's index. ReportRun loses trigger and its four cliff columns,
with their check. BudgetWindowRun loses trigger. ReportRunTrigger and
ReferencedUser have no other use, so they go too.

No statement named these columns already, apart from the models
themselves, so the models work on a database that still has them, where
new rows leave them null, and on one without them. The test fixtures stop
writing them so they run against both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture selected every column the model maps, so it would fail on a
database without one of them, whichever release it tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_tables counted eight enum columns, two of them the trigger columns.
test_mapped_classes repeated the foreign key total test_tables already
checks along with the keys themselves, so it no longer states one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two completion tests lost what they checked when the report run's
cliff columns went, so they now check that no cliff_impacts row appears.
Also drop two names that came from the removed trigger values and the
remaining notes that shared rows hold no label, user or trigger, which
no longer describe any column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With no cliff_impacts row seeded, the two completion tests could not see
an update of one, and the Modal worker role can't insert one. They now
seed a pending result for each side and check it stays unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant